Interpretable counting in visual question answering

ABSTRACT

Approaches for interpretable counting for visual question answering include a digital image processor, a language processor, and a counter. The digital image processor identifies objects in an image, maps the identified objects into an embedding space, generates bounding boxes for each of the identified objects, and outputs the embedded objects paired with their bounding boxes. The language processor embeds a question into the embedding space. The scorer determines scores for the identified objects. Each respective score determines how well a corresponding one of the identified objects is responsive to the question. The counter determines a count of the objects in the digital image that are responsive to the question based on the scores. The count and a corresponding bounding box for each object included in the count are output. In some embodiments, the counter determines the count interactively based on interactions between counted and uncounted objects.

RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication Ser. No. 62/578,379 filed on Oct. 27, 2017 and entitled“Interpretable Counting in Visual Question Answering,” which isincorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to the generation ofinterpretable counting results for questions related to image content.

BACKGROUND

Visual question answering (VQA) is a benchmark to test forcontext-specific reasoning about complex images. One aspect of visualquestion answering is related to the answering of counting questions(also known as “How Many” questions) that are related to identifyingdistinct scene elements or objects that meet some criteria embodied inthe question and counting the objects.

Accordingly, it would be advantageous to have systems and methods forcounting the objects in images that satisfy a specified criteria.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of a method of counting objects in animage according to some embodiments.

FIG. 2 is a simplified diagram of an image and a counting questionaccording to some embodiments.

FIG. 3 is a simplified diagram of a computing device according to someembodiments.

FIG. 4 is a simplified diagram of a system for counting objects in animage according to some embodiments.

FIG. 5 is a simplified diagram of another embedding and scoring moduleaccording to some embodiments.

FIGS. 6 and 7 are simplified diagrams of counters according to someembodiments.

FIG. 8 is a simplified diagram of a method for counting objects in animage according to some embodiments.

FIG. 9 is a simplified diagram of a method for counting objects matchinga question according to some embodiments.

FIG. 10 is a simplified diagram of images and counting questionsaccording to some embodiments.

FIG. 11 is a simplified diagram of counting results according to someembodiments.

FIGS. 12 and 13 are simplified diagrams of performance results forcounting objects in images according to some embodiments.

FIGS. 14A and 14B are simplified diagrams of counting grounding qualityaccording to some embodiments.

In the figures, elements having the same designations have the same orsimilar functions.

DETAILED DESCRIPTION

Context specific reasoning, including context specific reasoningregarding the content of images, is an important problem in machineintelligence and learning applications. Context specific reasoning mayprovide valuable information for use in the interpretation of medicalimages, diagnostic systems, autonomous vehicles, and/or the like. Onetype of context specific reasoning is the determination of how many(e.g., a count) objects in an image that meet a specified criteria. Inaddition to determining the count of the number of objects in the image,it may also be useful to have access to additional information allowingthe results of the counting to be further verified or interpreted byseeing which objects are counted. The information regarding whichobjects are counted help identify instances where the count is correct,but for the wrong reasons (e.g., the objects that did not meet thespecified criteria are counted and object that met the criteria arenot). The information regarding which object is counted may also beuseful in identifying regions in the image that are to be furtherprocessed. And although the various embodiments described within thisdisclosure are generally related to answering natural language countingquestions (e.g., questions in English) about the content of an image, itis understood that the described embodiments may be adapted to othercontext-specific reasoning applications, such as those related to video,questions in forms other than natural language, questions provided inaudio form, and/or the like.

FIG. 1 is a simplified diagram of a method of counting objects in animage according to some embodiments. One or more of the processes110-140 of method 100 may be implemented, at least in part, in the formof executable code stored on non-transitory, tangible, machine-readablemedia that when run by one or more processors may cause the one or moreprocessors to perform one or more of the processes 110-140. And althoughFIG. 1 implies an ordering to processes 110-140 it is understood thatthe processes may be performed in other orders. In some examples,processes 110 and 120 may be performed in any order and/or concurrently.Method 100 is described in the context of FIG. 2, which is a simplifieddiagram of an image 210 and a counting question 220 according to someembodiments.

At a process 110, an image is received. Image 210 is an example of animage that may be received during process 110. In some examples, theimage may be received as a file in an image format, such as GIF, JPEG,bitmap, and/or the like. In some examples, the image may be receivedfrom an imaging device, such as a camera, a video camera, and/or thelike.

At a process 120, a natural language question is received. In someexamples, the question is a counting question, such as question 220,which asks: “How many animals are there?” Other questions may include“how many trees are there?” and/or the like. In some examples, thequestion may be received as a file in a text-interpretable format, suchas text, XML, JSON, a word processing format, and/or the like. In someexamples, the question may be typed in by a user, transcribed from anaudio sample, and/or the like.

At a process 130, the object(s) from the image matching the question arecounted. Specific examples of counting systems and methods are describedin further detail below.

At a process 140, the results of the counting are reported and/or outputin interpretable form. Counting result 230 is an example of a reportedcount. In some examples, the count is reported as a numeric value, intextual form, as audio output, and/or the like. Additional informationregarding the results of the counting is also presented. As an example,bounding boxes, such as bounding boxes 240 may be overlaid on image 210to identify the objects that resulted in the counting result 230. Otherexamples of additional information may include coordinates of thebounding boxes, use of other shapes, extracted sub-images, and/or thelike. In some examples, bounding boxes 240 allow the user to interpretand/or verify that counting result 230 is correct for the right reasons.

FIG. 3 is a simplified diagram of a computing device 300 according tosome embodiments. As shown in FIG. 3, computing device 300 includes aprocessor 310 coupled to memory 320. Operation of computing device 300is controlled by processor 310. And although computing device 300 isshown with only one processor 310, it is understood that processor 310may be representative of one or more central processing units,multi-core processors, microprocessors, microcontrollers, digital signalprocessors, field programmable gate arrays (FPGAs), application specificintegrated circuits (ASICs), graphics processing units (GPUs) and/or thelike in computing device 300. Computing device 300 may be implemented asa stand-alone subsystem, as a board added to a computing device, and/oras a virtual machine.

Memory 320 may be used to store software executed by computing device300 and/or one or more data structures used during operation ofcomputing device 300. Memory 320 may include one or more types ofmachine readable media. Some common forms of machine readable media mayinclude floppy disk, flexible disk, hard disk, magnetic tape, any othermagnetic medium, CD-ROM, any other optical medium, punch cards, papertape, any other physical medium with patterns of holes, RAM, PROM,EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any othermedium from which a processor or computer is adapted to read.

Processor 310 and/or memory 320 may be arranged in any suitable physicalarrangement. In some embodiments, processor 310 and/or memory 320 may beimplemented on a same board, in a same package (e.g.,system-in-package), on a same chip (e.g., system-on-chip), and/or thelike. In some embodiments, processor 310 and/or memory 320 may includedistributed, virtualized, and/or containerized computing resources.Consistent with such embodiments, processor 310 and/or memory 320 may belocated in one or more data centers and/or cloud computing facilities.

As shown, memory 320 includes a counting module 330 that may be used toimplement and/or emulate any of the counting systems described furtherherein and/or to implement any of the methods described further herein.In some examples, counting module 330 may be used to count the objectsin an image that match criteria included in a question about the image.In some examples, counting module 330 may also handle the iterativetraining and/or evaluation of a counting system used to count theobjects in an image that match criteria included in a question about theimage. In some examples, memory 320 may include non-transitory,tangible, machine readable media that includes executable code that whenrun by one or more processors (e.g., processor 310) may cause the one ormore processors to perform the counting methods described in furtherdetail herein. In some examples, counting module 330 may be implementedusing hardware, software, and/or a combination of hardware and software.As shown, computing device 300 receives an image 340 and a countingquestion 350 about image 340, which are provided to counting module 330,which generates a count 360 of the objects from image 340 matching acriteria from question 350 and a set of object outlines 370 including anoutline of each of the counted objects.

FIG. 4 is a simplified diagram of a system 400 for counting objects inan image according to some embodiments. According to some embodiments,system 400 may be capable of performing method 100, may be implementedby counting module 330, and/or emulated by counting module 330. As shownin FIG. 4, system 400 receives an image 410. In some examples, image 410may be received as a file in an image format, such as GIF, JPEG, bitmap,and/or the like. In some examples, image 410 may be received from animaging device, such as a camera, a video camera, and/or the like.

Image 410 is provided to an image processing module 420. In someexamples, image processing module 420 processes image 410 and generatesa set of coordinates xy. Each entry in the coordinates xy identifies arectangular bounding box (e.g., (x min, y min, x max, y max)∈

⁴) identifying a candidate object within image 410. And althoughcoordinates xy includes rectangular bounding boxes it is understood thatother bounding shapes (e.g., circles) and/or region identifyingapproaches are possible. Image processing module 420 further generatesobject embeddings v that encode characteristics of each of the objectsidentified by a corresponding rectangular bounding box in coordinates xyinto an embedding space. In some examples, image processing module 420includes a Faster R-CNN that generates coordinates xy and objectembeddings v. The Faster R-CNN is described in further detail in Ren, etal., “Faster R-CNN: Towards Real-Time Object Detection with RegionProposal Networks,” Advances in Neural Information Processing Systems28, 2015, which is incorporated by reference herein. In some examples,each of the object embeddings maps information from image 410 associatedwith a corresponding rectangular bounding box into a vector v_(i)∈

²⁰⁴⁸. In some examples, the Faster R-CNN is pre-trained as is describedin Anderson, et al., “Bottom-Up and Top-Down Attention for ImageCaptioning and VQA,” Computer Vision and Pattern Recognition Conference,2017, which is incorporated by reference herein. In some examples, thenumber of objects included in coordinates xy and object embeddings v isfixed at and/or limited to 1024.

System 400 further receives a question 430. In some examples, question430 is a counting question, such as question 220. In some examples, thequestion may be received as a file in an text-interpretable format, suchas text, XML, JSON, a word processing format, and/or the like. In someexamples, the question may be typed in by a user, transcribed from anaudio sample, and/or the like. In some examples, question 430 may beexpressed in natural language (e.g., English) with each word encodedinto a vector x_(i)∈

³⁰⁰. In some examples, the word encodings are the GloVe encodingsdescribed in further detail in Pennington, et al., “Global Vectors forWord Representation,” Proceedings of the 2014 Conference on EmpiricalMethods in Natural Language Processing (EMNLP), pp. 1532-1543, 2014,which is incorporated by reference herein.

The encoded question {x₁, x₂, . . . , x_(m)} is provided to a languageprocessing module 440, which generates as output question embedding qthat encodes semantic characteristics of question 430. In some examples,language processing module 440 includes a recurrent long-term short-termmemory (LSTM) network, which generates question embedding q according toEquation 1. LSTM networks for language processing are described infurther detail in Hochreiter, et al. “Long Short-Term Memory,” Journalof Neural Computation, 9(8), pp. 1735-1780, 1997, which is incorporatedby reference herein. In some examples, question embedding q is the samesize and in the same embedding space as each of the object embeddingsv_(i) in object embeddings v.h ^(t)=LSTM(x ^(t) ,h ^(t−1))q=h ^(T)  Equation 1

Question embedding q from language processing module 440 and objectembeddings v from image processing module 420 are provided to a scorer450. Scorer 450 generates a scoring vector s, which includes a scores_(i)∈

^(n) for each of the object embedding v_(i) in object embeddings v. Eachscore, s_(i) indicates how well the corresponding object embedding v_(i)matches question embedding q. In some examples, each score s_(i) isdetermined according to Equation 2, where f^(S) is a layer of gated tanh units (GTUs), which maps a concatenation of question embedding q andobject embedding v_(i) to score s_(i). GTUs are described in van denOord, et al. “Conditional Image Generation with PixelCNN Decoders,”Advances in Neural Information Processing Systems 29, 2016 and the useof GTUs for scoring are described in Anderson, et al., “Bottom-Up andTop-Down Attention for Image Captioning and VQA,” Computer Vision andPattern Recognition Conference, 2017, each of which is incorporated byreference herein.s _(i) =f ^(s)([q,v _(i)])  Equation 2

Coordinates xy, object embeddings v, score vector s, and questionembedding q are provided to a counter 460. Counter 460 generates a count470 and a set of outlines 480 of the objects selected for counting.Various embodiments of counter 460 are described in further detailbelow.

FIG. 5 is a simplified diagram of another embedding and scoring module500 according to some embodiments. In some embodiments, embedding andscoring module 500 may be used as an alternative to language processingmodule 440 and scorer 450 of system 400. As shown in FIG. 5, embeddingand scoring module 500 receives a question 510, which is provided tolanguage processing module 520 to generate question embedding q. In someexamples, question 510 and language processing module 520 aresubstantially similar to question 430 and language processing module440, respectively.

Embedding and scoring module 500 further receives a caption 530, whichis provided to a captioning module 540. In some examples, caption 530corresponds to a textual description of the content of a region withinthe same image used to generate object embeddings v. In some examples,the same image may correspond to image 410. In some examples, captioningmodule 540 may include an LSTM-based architecture similar to languageprocessing module 520.

Question embedding q and the caption embedding as well as objectembeddings v (e.g., from image processing module 420) are provided to ascorer 550, which generates scoring vector s. In some examples, scorer550 is similar to scorer 450. In some examples, scorer 550 may alsoprovide caption grounding in addition to scoring vector s. In someexamples, caption grounding attempts to match a region of an image to acaption, which is similar to VQA. In some examples, multiple captions530 are provided and scorer 550 generates a scoring of how well eachobject in object embeddings v matches each of the embedded captions. Insome embodiments, the inclusion of the caption grounding in embeddingand scoring module 500 allows for better training of scorer 550 incomparison to scorer 450 without the caption grounding.

FIG. 6 is a simplified diagram of a counter 600 according to someembodiments. In some embodiments, counter 600 may be used as counter460. In some embodiments, counter 600 may be referred to as a SoftCountcounter 600. As shown in FIG. 6, counter 600 receives a scoring vectors, which may be consistent with scoring vector s from FIGS. 4 and 5.Scoring vector s is provided to a weighting module 610 which weights thescores s_(i) in scoring vector s according to trainable weights W. Atrainable bias b is further added to the weighted scores using a summingunit 620. The resulting total is provided to a transfer function 630,which generates a count C. Counter 600 generates count C according toEquation 3, where i indicates each of the candidate objects in objectembeddings v, which may correspond to the question. In some examples, σis a sigmoid transfer function, such as softmax, log-sigmoid, hyperbolictangent sigmoid, and/or the like. In some examples, counter 600generates a trainable weighted sum of the scores s_(i) of each of theobjects in the image, and the result is rounded to provide a wholenumber result.

$\begin{matrix}{C = {{round}( {\sum\limits_{i}\;{\sigma( {{Ws}_{i} + b} )}} )}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

Counter 600 further outputs the output of summing unit 620 as a matchlevel and passes through coordinates xy as the object outlines as outputby counter 600. By passing coordinates xy of each of the detectedobjects along with the match level, partial interpretability of counter600 is possible because the match level of each of the objects providesan indication of the relative strength of each of the objects that matchthe question. In some embodiments, scoring vector s may be substitutedfor the output of summing unit 620 to provide the match level.

In some examples, counter 600 may be trained via back propagation so asto minimize the Huber loss L_(i) of the counting results according toEquation 4, where C^(GT) is the ground truth count for the image andquestion provided for counting.

$\begin{matrix}{L_{i} = \{ {{\begin{matrix}{0.5e^{2}} & {e \leq 1} \\{e - 0.5} & {e > 1}\end{matrix}e} = {{C - C^{GT}}}} } & {{Equation}\mspace{14mu} 4}\end{matrix}$

FIG. 7 is a simplified diagram of another counter 700 according to someembodiments. In some embodiments, counter 700 may be used as counter460. In some embodiments, counter 700 may be referred to as aninterpretable reinforcement learning counter (IRLC) 700. As shown inFIG. 7, counter 700 receives a scoring vector s, which may be consistentwith scoring vector s from FIGS. 4 and 5. Scoring vector s is providedto a weighting module 710 which weights the scores s_(i) in scoringvector s according to trainable weights W. A trainable bias b is furtheradded to the weighted scores using a summing unit 720. The resultingtotal is provided to a transfer function 730. In some examples,weighting module 710, summing unit 720, and transfer function 730generate an output according to Equation 5. In some examples, f maycorrespond to the pure linear function, the softmax function, and/or thelike. The resulting total is provided to a transfer function 730, whoseoutput is provided to a logits module 740 as an initial set of logitvalues k⁰. Each logit value is an indicator of how well thecorresponding object is likely to satisfy the counting question beingconsidered by counter 700.k ⁰ =f(Ws+b)  Equation 5

Counter 700 further includes an interaction module 750, which receivescoordinates xy and object embeddings v (e.g., from image processingmodule 420) as well as question embedding q (e.g., from languageprocessing module 440 and/or language processing module 520).Interaction module 750 further receives information on each objectpreviously selected by object selector 760. Interaction module 750provides updates to the logit values k being maintained by logits module740 that are caused by the selection of an object to be counted byobject selector 760. In some examples, interaction module 750 maintainsa matrix of interaction terms ρ_(ij) according to Equation 6, where [,]is vector concatenation, {circumflex over (v)}_(i) is a normalizedversion of object embedding v_(i) for object i, xy_(i) corresponds tothe bounding box for object i, IoU_(ij), O_(ij), and O_(ji) are overlapstatistics for objects i and j. In some examples, f^(ρ) corresponds to atrainable two-layer perceptron network using rectified linear unit(ReLU) activation in the hidden layers. In some examples, becauseinteraction module 750 considers similarities between the objectembeddings v and overlaps between the rectangular bounding boxes of twoobjects, interaction module 750 is able to learn not to count an objecttwice even when scoring vector s includes two potential objects i and jwith high scores s_(i) and s_(j) and an overlap in area.

$\begin{matrix}{{\rho_{ij} = {f^{\rho}( \lbrack {{Wq},{{\hat{v}}_{i}^{T}{\hat{v}}_{j}},{xy}_{i},{xy}_{j},{IoU}_{ij},O_{ij},O_{ji}} \rbrack )}}{{IoU}_{ij} = \frac{{area}( {{xy}_{i}\bigcap{xy}_{j}} )}{{area}( {{xy}_{i}\bigcup{xy}_{j}} )}}{O_{ij} = \frac{{area}( {{xy}_{i}\bigcap{xy}_{j}} )}{{area}( {xy}_{i} )}}{O_{ji} = \frac{{area}( {{xy}_{i}\bigcap{xy}_{j}} )}{{area}( {xy}_{j} )}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

Object selector 760 receives the logit values k from logits module 740and determines whether another object is to be counted or that countingof the objects matching the criteria of the question is complete. Objectselector 760 operates in an iterative fashion and, with each iteration,object selector 760 either selects the object with the highest logitvalue or terminates the counting. When object selector 760 selects anobject for counting, the selected object is passed to interaction module750 to determine how selection of that object affects the logit values kof other objects being considered for counting. In some examples, objectselector 760 selects an object to count using Equation 7, where a^(t)corresponds to the t-th object selected for counting and ζ correspondsto a learnable threshold that indicates that no current logit value ishigh enough to represent a countable object. With each new question,object selector 760 is initialized so that no objects are selected.a ^(t)=argmax[k ^(t),ζ]  Equation 7

Once an object is selected for counting, logits module 740 uses theinteraction values from interaction module 750 to update the logitvalues according to Equation 8, where ρ(a^(t),•) corresponds to the rowof interaction matrix ρ corresponding to the selected object a^(t).k ^(t+1) =k ^(t)+ρ(a ^(t),•)  Equation 8

In some embodiments, counter 700 may be trained using reinforcementlearning. In some examples, the REINFORCE learning rule is used. TheREINFORCE learning rule is described in Williams, “Simple StatisticalGradient-following Methods for Connectionist Reinforcement Learning,”Machine Learning, 8:229-256, 1992, which is incorporated by referenceherein. In some examples, the reward R is determined according to avariation of policy gradient called self-critical sequence training.Self-critical sequence training is described in Rennie, et al.,“Self-critical Sequence Training for Image Captioning,” Computer Visionand Pattern Recognition Conference, 2017, Anderson, et al., “Bottom-Upand Top-Down Attention for Image Captioning and VQA,” Computer Visionand Pattern Recognition Conference, 2017, and Paulus, et al, “A DeepReinforced Model for Abstractive Summarization,” arXiv, 2017, each ofwhich is incorporated by reference herein. In some examples, thecounting loss L_(C) may be determined according to Equation 9, wherep^(t) is approximately equal to a^(t), E^(greedy) is the baseline counterror obtained by greedy object selection (e.g., selecting the objectwith the highest logit k^(t) using Equation 7) and E is the baselinecount error obtained by selecting the object randomly (e.g., sampling)based on the probability values p^(t), with training being performed toreinforce the selection of the randomly selected objects.

$\begin{matrix}{{L_{C} = {{- R}{\sum\limits_{t}\;{{\log( p^{t} )}a^{t}}}}}{R = {E^{greedy} - E}}{E = {{C - C^{GT}}}}{p^{t} = {{softmax}( \lbrack {k^{t},\xi} \rbrack )}}} & {{Equation}\mspace{14mu} 9}\end{matrix}$

In some examples, additional errors are included in the loss function asshown in Equation 10, where H is the policy entropy function and L₁ isthe Huber loss from Equation 4. In some examples, λ₂=λ₃=0.005λ₁.

$\begin{matrix}{{L_{total} = {{\lambda_{1}L_{C}} + {\lambda_{2}P_{H}} + {\lambda_{3}P_{I}}}}{P_{H} = {- {\sum\limits_{t}\;{H( p^{t} )}}}}{P_{I} = {\sum\limits_{i \in {\{{a^{1},\ldots,a^{t}}\}}}\;{\frac{1}{N}{\sum\limits_{j}\; L_{1}}}}}} & {{Equation}\mspace{14mu} 10}\end{matrix}$

FIG. 8 is a simplified diagram of a method 800 of training a countingsystem according to some embodiments. One or more of the processes810-890 of method 800 may be implemented, at least in part, in the formof executable code stored on non-transitory, tangible, machine-readablemedia that when run by one or more processors (e.g., processor 310) maycause the one or more processors to perform one or more of the processes810-890. According to some embodiments, method 800 may be used to trainsystem 400, counter 600, and/or counter 700. According to someembodiments, method 800 may be performed in different ways than impliedby the ordering of FIG. 800. In some examples, processes 810 and 820 maybe performed and/or concurrently with processes 830 and 840. In someembodiments, process 870 is optional and may be omitted.

At a process 810, an image is received. In some examples, the image maycorrespond to image 410. In some examples, the image may be received asa file in an image format, such as GIF, JPEG, bitmap, and/or the like.In some examples, the image may be received from an imaging device, suchas a camera, a video camera, and/or the like.

At a process 820, object coordinates and object embeddings aregenerated. The object coordinates correspond to the border or eachcandidate object identified in the image received during process 810.The object embeddings include an encoding representing the content ofthe image from a sub-region identified by a corresponding border in thegenerated object coordinates. In some examples, the object coordinatesdescribe a rectangular bounding box. In some examples, the objectcoordinates and object embeddings may be generated using imageprocessing module 420. In some examples, the generated objectcoordinates may correspond to object coordinates xy and the generatedobject embeddings may correspond to object embeddings v.

At a process 830, a question is received. In some examples, the questionis consistent with question 430 and/or question 510. In some examples,the question may be a counting question, such as “how many animals arethere?”, “how many trees are there?”, and/or the like. In some examples,the question may be received as a file in a text-interpretable format,such as text, XML, JSON, a word processing format, and/or the like. Insome examples, the question may be typed in by a user, transcribed froman audio sample, and/or the like. In some examples, a ground truth countassociated with the question may also be received.

At a process 840, question embeddings are generated. The questionembeddings include an encoding of the question received during process830. In some examples, the question embeddings may be generated usinglanguage processing module 440 and/or language processing module 520. Insome examples, the generated question embeddings may correspond toquestion embeddings q.

At a process 850, a scoring vector is generated. The scoring vector isgenerated based on the object embedding generated during process 820 andthe question embeddings generated during process 840. The scoring vectorincludes a score indicating how well each of the objects in thegenerated object embeddings generated during process 820 matches thecriteria encoded in the question embeddings generated during process840. In some examples, the scoring vector may be generated by scorer 450and/or scorer 550. In some examples, the generated scoring vector maycorrespond to scoring vector s.

At a process 860, objects matching the question are counted. Theresulting count corresponds to the number of objects in the imagereceived during process 810 that appear to match the criteria from thequestion received during process 830. In some embodiments, the count maybe generated by weighting module 610, summing unit 620, and transferfunction 630 of SoftCount counter 600 according to Equation 3. In someembodiments, the count may be generated according to IRLC 700.

FIG. 9 is a simplified diagram of a method 800 for counting objectsmatching a question according to some embodiments. One or more of theprocesses 910-960 of method 900 may be implemented, at least in part, inthe form of executable code stored on non-transitory, tangible,machine-readable media that when run by one or more processors (e.g.,processor 310) may cause the one or more processors to perform one ormore of the processes 910-960. According to some embodiments, method 900is consistent with the method used by IRLC 700 to count objects matchingthe criteria in a question.

At a process 910, initial logit values are generated. Each of theinitial logit values corresponds to how well a corresponding objectmatches the criteria in the question. The initial logit values aregenerated based on a scoring vector, such as the scoring vectorgenerated during process 850. In some examples, the initial logit valuesare generated by weighting module 710, summing unit 720, and transferfunction 730 according to Equation 5.

At a process 920, a best candidate object is selected from those objectsnot yet selected and counted by method 900. The best candidate objectcorresponds to the not yet selected object having the highest logitvalue. In some examples, the highest logit value may be determinedaccording to Equation 7. In some examples, during training, twocandidate objects are selected based on Equation 7 and randomly based onthe p^(t) probabilities of Equation 9 with both selected objects beingused for separate counting results in parallel. In some examples,process 920 may further record which object is selected, such as bystoring an index value of the selected object. In some examples, process920 is performed by object selector 760.

At a process 930, it is determined whether the object counting is to beterminated. In some examples, object counting is terminated when none ofthe remaining unselected objects has a corresponding logit value that isgreater than a termination value, such as termination value ζ. In someexamples, process 920 may select termination value ζ instead of thelogit value of an unselected object according to Equation 7. In someexamples, process 930 is performed by object selector 760. When anobject is selected during process 920 instead of the termination value,the object is counted and the logit values are updated beginning with aprocess 940. When no unselected objects have a corresponding logit valuegreater than the termination value and the termination value is selectedduring process 920, the results of the counting are reported using aprocess 960.

At process 940, the selected object is counted. In some examples, theobject selected during process 920 may be counted by incrementing acounter that was initialized to zero before method 900 began. In someexamples, process 940 is performed by object selector 760.

At a process 950, the logit values are updated according to theirinteraction with the object selected during process 920. The logit valuefor each of the unselected objects is updated based on an interactionbetween each of the unselected objects and the object selected duringprocess 920. In some examples, the logit value of a first object fromthe unselected objects may be updated based on the question embeddings(e.g., the question embeddings generated during process 840),similarities between the object embeddings (e.g., the object embeddingsgenerated during process 820) of the first object and the objectselected during process 920, the coordinates (e.g., the coordinates xygenerated during process 820) of the first object and the objectselected during process 920, and overlap statistics between thecoordinates of the first object and the object selected during process920. In some examples, the logit values may be updated using logitsmodule 740 and interaction module 750 according to Equations 7 and 8.Once the logit values are updated by process 950, method 900 returns toprocess 920 to determine whether another object is selected and countedor the termination value is selected.

At process 960, the count and the outline(s) of the selected object(s)are reported. In some examples, the count is the value of the counterincremented during process 940. In some examples, when no objects arecounted (e.g., the counter is zero), a count of zero is reported and nooutlines are reported. In some examples, when the count is one orhigher, the one or more outlines correspond to the coordinates (e.g.,the coordinates xy generated during process 820) of each of the objectsselected by the counting iterations of process 920. In some examples,each of the one or more outlines may correspond to a bounding rectanglefor a sub-portion of the image that contains the corresponding object.In some examples, process 960 may determine which of the coordinates toreport based on the object indices stored during process 920. In someexamples, process 960 is performed by object selector 760.

The operation of method 900 is demonstrated in FIG. 10, which is asimplified diagram of images and counting questions according to someembodiments. More specifically, FIG. 10 shows the counting results, forfour image and question pairs 1010, 1020, 1030, and 1040, through eachiteration of IRLC 700 and method 900, where method 900 performs avariation of process 960 along with each performance of processes 940and 950. At t=0, the image is shown with the outlines of each of thecandidate objects identified by, for example, process 820. Each of thecoordinates is shown as the outline of a bounding rectangle around thecorresponding candidate object. In some examples, the opacity and/orintensity of the rectangle is rendered to indicate the strength of thecorresponding logit value generated during process 910. At t=1, a firstobject has been selected by process 920 and its corresponding boundingrectangle is rendered in a different color to indicate a selected andcounted object. An example of the bounding rectangle for a selected andcounted object is shown as a rectangle 1015 for image and question pair1010. With the selection of the first object at t=1, the logit valuesare updated by process 950 and reflected by the changes in opacityand/or intensity of the bounding rectangles of the remaining candidateobjects. At t=2, a second object is selected and counted, the boundingrectangle of the second object is updated as well the logit values basedon the selection of the second object. For example, the t=2 image withbounding rectangle overlays for image and question pair 1010 shows thatthe logit values of most of the unselected objects have beensignificantly reduced. At C=3, a third object is selected and counted,the bounding rectangle of the third object is updated as well the logitvalues based on the selection of the third object. As none of theremaining unselected objects have a logit value greater than thetermination value, method 900 concludes with a count value of 3 and theoutlines (e.g., the bounding rectangles) of the counted objects isreported. Image and question pair 1040 indicates the further value ofmaking the counting results interpretable by reporting the coordinatesof the selected and counted objects. Even though method 900 obtained thecorrect count of 3 for image and question pair 1040, the reportedcoordinates indicate that the wrong three objects were counted as object1042 is not an egg and object 1044 is an egg that was not counted. Thus,the interpretability provided by reporting the outlines of the countingobjects provides the additional ability to differentiate betweencounting results for the correct reasons and counting results that arecorrect, but for the wrong reasons.

Referring back to FIG. 8, at an optional process 870, object outlinesare output. In some embodiments, when process 860 used SoftCount counter600, the object outlines may correspond to each of the objects in theobject coordinates determined during process 820. In some examples, eachof the outlines may also be associated with a corresponding metricindicating how well the corresponding object matched the criteria of thequestion received during process 830. In some examples, the metricindicating how well the corresponding object matched the criteria may bebased on the corresponding score value generated during process 850and/or the corresponding value generated by function a of Equation 3. Insome embodiments, when process 860 used IRLC 700 and/or method 900, theobject outlines may correspond to each of the objects selected by objectselector 760 and/or process 920. In some examples, the object outlinesmay be output by overlaying the object outlines on the image receivedduring process 810, for example such as is shown by bounding boxes 240in FIG. 2. In some examples, when process 860 used SoftCount counter600, each of the object outlines may be drawn with an opacity and/orintensity based on the corresponding metric indicating how well thecorresponding object matched the criteria of the question receivedduring process 830. In some examples, the opacity and/or intensity maybe set according to the amount the corresponding object contributed tothe count in Equation 3. In some examples, the object outlines areuseable to interpret and/or validate the count determined during process860.

At a process 880, counting error is determined. The counting errorindicates how well the count determined during process 860 matches aground truth count for the image. In some embodiments, when process 860uses SoftCount counter 600, the counting error may be determinedaccording to Equation 4. In some embodiments, when process 860 uses IRLC700 and/or method 900, the counting error may be determined according toEquation 9 and/or Equation 10 using both counting results based on thedifferent candidate objects selected during process 920.

At a process 890, the counting is trained based on the counting errordetermined during process 890. In some examples, the counting error maybe back propagated to adjust the various weights, biases, and/or thelike in counter 460, counter 600, and/or counter 700. In someembodiments, the counter error may be further back propagated to imagingprocessing module 420, language processing module 440, and/or languageprocessing module 520 to provide end-to-end training of the countingsystem.

According to some embodiments, method 800 may be repeatedly used as partof a larger training process where a large number of training images andcounting questions are each presented in turn to the counting systemwith training for each pair of image and counting question occurringaccording to method 800. In some examples, each of the images andcounting questions may be presented multiple times during the trainingwith each presentation of the set of images and counting question pairscorresponding to a training epoch. In some examples, the training mayoccur according to the adaptive moment estimation (ADAM) algorithm. TheADAM algorithm is described in Kingma, et al., “A Method for StochasticOptimization,” 3rd International Conference for LearningRepresentations, 2015, which is incorporated by reference herein. Insome examples, when the ADAM algorithm is used to train SoftCountcounter 600, the learning rate is set to 3×10⁻⁴ and decayed by 0.8 witheach epoch after the training accuracy plateaus. In some examples, whenthe ADAM algorithm is used to train IRLC 700, the learning rate is setto 5×10⁻⁴ and is decayed by 0.99999 with each epoch.

According to some embodiments, method 800 may be modified to generateinterpretable counts for images using, for example, system 400. In someexamples, an image and question, including an image and question withouta ground truth count, may be processed according to processes 810-870 togenerate a count and image outlines corresponding to the number ofobjects in the image that match the criteria of the question.

According to some embodiments, method 800 may be further adapted toperform caption grounding. In some examples, a caption (e.g., caption530) may be received using a process similar to process 830. In someexamples, a caption embedding may be generated by a captioning module(e.g., captioning module 540) using a process similar to process 840. Insome examples, a scorer (e.g., scorer 550) may be used to determinewhich how well the caption corresponds to each of the objects determinedduring process 820.

FIG. 11 is a simplified diagram of counting results according to someembodiments. More specifically, FIG. 11 shows the counting results, forfive image and question pairs 1110-1150 along with their ground truthvalues and the results of three different counters (the SoftCountcounter 600, an UpDown counter, and IRLC 700). The UpDown counter is anattention-based counter used as a baseline to comparatively assess thecapabilities of SoftCount counter 600 and IRLC 700. The UpDown counteris described in Anderson, et al., “Bottom-Up and Top-Down Attention forImage Captioning and VQA,” Computer Vision and Pattern RecognitionConference, 2017, which is incorporated by reference herein. Each of theresults for SoftCount counter 600 is shown with its count value abovethe image and the opacity of the various outlines corresponds to theamount the corresponding object contributed to the count in Equation 3,where the contribution varies between (0=transparent) and (1=opaque).Each of the results for the UpDown counter is shown with its count valueabove the image and the opacity of the various outlines corresponds tothe attention focus generated by the UpDown counter. Each of the resultsfor IRLC 700 is shown with its count value above the image and theoutlines indicate the counted objects.

According to some embodiments, the counting results of individual imageand question pairs, such as those in FIG. 11, provide an indication ofthe capabilities of each of the SoftCount counter 600, the UpDowncounter, and IRLC 700. However, other metrics provide additionalinsight.

According to some embodiments, one evaluation metric is countingaccuracy. In some examples, count accuracy is derived from the countsprovided by a panel of 10 human reviewers when presented with each imageand question pair. In some examples, a counting answer is consideredcorrect if at least three of the human viewers provided that count withthe accuracy of each possible counting answer being determined accordingto Equation 11. In some examples, the accuracy results may be averagedfor multiple panels of reviewers generated by considering each of thepossible combinations of human reviewers.

$\begin{matrix}{{{Accuracy}(a)} = {\min\lbrack {\frac{\#\mspace{14mu}{humans}\mspace{14mu}{that}\mspace{14mu}{answered}\mspace{14mu} a}{3},1} \rbrack}} & {{Equation}\mspace{14mu} 11}\end{matrix}$

In some examples, the accuracy value provided by Equation 11 may be usedto assess the accuracy of the counting values reported by SoftCountcounter 600, the UpDown counter, and IRLC 700 with the accuracy resultsaveraged across each image and question pair evaluated by eachrespective counter.

According to some embodiments, another evaluation metric is root meansquare error (RMSE) determined across each image and question pairevaluated by each respective counter. In some examples, RMSE provides abetter indication of overall counting ability as it also accounts forhow close an incorrect count is to the ground truth count. In someexamples, RMSE is determined according to Equation 12, where N is thenumber of image and question pairs evaluated by each respective counterand a lower RMSE is desired.

$\begin{matrix}{{RMSE} = \sqrt{\frac{1}{N}{\sum\limits_{i}\;( {C_{i}^{GT} - C_{i}} )^{2}}}} & {{Equation}\mspace{14mu} 12}\end{matrix}$

According to some embodiments, another evaluation metric is groundingquality. The grounding quality metric captures how relevant each of theobjects counted by a counter are to the criteria included in thequestion. The grounding quality metric is derived from the objectcategories assigned to various objects in each image in the COCOdataset. Each of the categories x is assigned a GloVe embeddingGlove(x)∈

³⁰⁰. In some examples, each object in a test set may be analyzed byfirst assigning one of the COCO categories or background to each of thecandidate objects used for counting (e.g., each of the objects in objectembeddings v) by identifying the object in the COCO dataset with thelargest intersection over union (IoU) overlap (see Equation 6) with acandidate object. And, if the IoU is greater than 0.5, the candidateobject is assigned to the COCO category of the object in the COCOdataset, otherwise the candidate object is assigned as background. TheCOCO category assigned to each object i from image m is denoted k_(i)^(m). Each COCO category present in image m is then converted to aquestion (e.g., the category “car” is converted to “how many cars arethere?”). The image is then examined using a counter, such as SoftCountcounter 600 and/or IRLC 700 and a count for the image m andcategory/question q is determined according to Equation 13, where w_(i)^((m,q)) is the count value given by the counter to candidate object iin image m and N^(m) is the number of candidate objects in image m.

$\begin{matrix}{C^{{({m,q})})} = {\sum\limits_{i}^{N^{m}}\; w_{i}^{({m,q})}}} & {{Equation}\mspace{14mu} 13}\end{matrix}$

A weighted sum is then generated that accounts for the semanticsimilarity between the category of the question q and the category ofthe candidate object according to Equation 14, where semantic similarityis determined based on the dot product between the GloVe embeddings ofthe category assigned to the candidate object k_(i) ^(m) and thecategory of the question q. When the candidate object k_(i) ^(m) isassigned to the background, its embedding is a vector of zeros.

$\begin{matrix}{s^{{({m,q})})} = {\sum\limits_{i}^{N^{m}}\;{w_{i}^{({m,q})}( {{{GloVe}( k_{i}^{m} )}^{T}{{GloVe}(q)}} }}} & {{Equation}\mspace{14mu} 14}\end{matrix}$

An aggregate caption grounding metric is then determined for each COCOcategory across each of the images by accumulating the weighted sums ofEquation 14 from individual images and normalizing across the totalcount according to Equation 13 from each of the images as shown inEquation 15.

$\begin{matrix}{s^{{(q)})} = \frac{\sum\limits_{m}\; s^{{({m,q})})}}{\sum\limits_{m}\; C^{{({m,q})})}}} & {{Equation}\mspace{14mu} 15}\end{matrix}$

FIGS. 12 and 13 are simplified diagrams of performance results forcounting objects in images according to some embodiments. FIG. 12 showsthe accuracy and RMSE of a Guess1 counter, a LSTM counter, SoftCountcounter 600, the UpDown counter, and IRLC 700 for image and questionpairs from the HowMany-QA test set. The Guess1 counter always guessesthat the count is 1 (the most common count in the HowMany-QA test set)and the LSTM counter predicts the count based on a linear projection ofthe question embeddings (e.g., question embedding q). Results areadditionally shown for SoftCount counter 600, the UpDown counter, andIRLC 700 when the counter includes the benefit of the caption groundingtraining from FIG. 5 (value not in parentheses) in comparison to thecounter without benefit of the caption grounding training from FIG. 5(value in parentheses). As shown in FIG. 12, IRLC 700 demonstrates boththe best accuracy and RMSE when caption ground is both included andomitted. And although SoftCount counter 600 demonstrates a loweraccuracy than the UpDown counter and IRLC 700 (e.g., it is wrong moreoften) the RMSE of SoftCount counter 600 is as good as IRLC 700indicating that SoftCount counter 600 does not tend to be too wrong whenit is incorrect.

FIG. 13 shows the differences in accuracy and RMSE for SoftCount counter600, the UpDown counter, and IRLC 700 when each of these counters istrained using different datasets. As shown, each counter is trainedusing the VQA HowMany-QA dataset alone (VQA only) and with both the VQAHowMany-QA dataset and the Visual Genome QA dataset. As expected, eachof the counters shows an improvement in both accuracy and RMSE with moretraining. However, IRLC 700 demonstrates both the best accuracy and RMSEof each of the three counters and additionally shows the least fall-offin both accuracy and RMSE when trained without the additional knowledgeprovided by the Visual Genome QA dataset. SoftCount counter 600 againdemonstrate a poorer accuracy, but a better RMSE than the UpDowncounter.

FIGS. 14A and 14B are simplified diagrams of counting grounding qualityaccording to some embodiments. FIGS. 14A and 14B show the groundingquality of both SoftCount counter 600 and IRLC 700 for each of the COCOcategories. As is shown, IRLC 700 generally demonstrates a highergrounding quality metric than SoftCount counter 600 over mostcategories.

This description and the accompanying drawings that illustrate inventiveaspects, embodiments, implementations, or applications should not betaken as limiting. Various mechanical, compositional, structural,electrical, and operational changes may be made without departing fromthe spirit and scope of this description and the claims. In someinstances, well-known circuits, structures, or techniques have not beenshown or described in detail in order not to obscure the embodiments ofthis disclosure. Like numbers in two or more figures represent the sameor similar elements.

In this description, specific details are set forth describing someembodiments consistent with the present disclosure. Numerous specificdetails are set forth in order to provide a thorough understanding ofthe embodiments. It will be apparent, however, to one skilled in the artthat some embodiments may be practiced without some or all of thesespecific details. The specific embodiments disclosed herein are meant tobe illustrative but not limiting. One skilled in the art may realizeother elements that, although not specifically described here, arewithin the scope and the spirit of this disclosure. In addition, toavoid unnecessary repetition, one or more features shown and describedin association with one embodiment may be incorporated into otherembodiments unless specifically described otherwise or if the one ormore features would make an embodiment non-functional.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the breadth and scopeof the present application should not be limited by any of theembodiments described herein, but should be defined only in accordancewith the following and later-submitted claims and their equivalents.

What is claimed is:
 1. A system for counting objects in a digital image,the system comprising: a digital image processor that identifies objectsin an image, maps the identified objects into an embedding space,generates bounding boxes for each of the identified objects, and outputsthe embedded objects paired with their bounding boxes; a languageprocessor that embeds a question into the embedding space; a scorer thatdetermines scores for the identified objects, each respective scoredetermines how well a corresponding one of the identified objects isresponsive to the question; a counter that determines a count of theobjects in the digital image that are responsive to the question basedon the scores; and the system outputs the count and a correspondingbounding box for each object included in the count.
 2. The system ofclaim 1, wherein the counter: determines whether to add a firstidentified object to the count; iteratively applies an interactionfilter to remaining identified objects to determine an uncounted objectthat is most likely responsive to the question based on each previouslycounted object; with each iteration, determines whether to add saiduncounted object to the count; and determines when to terminateprocessing objects in the image.
 3. The system of claim 2, wherein thecounter determines the first identified object to count by identifyingwhich of the identified objects has a highest measure of match to theembedding of the question.
 4. The system of claim 3, wherein todetermine when to terminate processing objects in the image, the counterdetermines whether the highest measure of match is less than atermination value.
 5. The system of claim 2, wherein the counterdetermines the interaction filter for a first unselected object of theidentified objects based on the embedding of the question, the embeddedobject corresponding to the first identified object, the embedded objectcorresponding to the first unselected object, and overlaps between thebounding boxes of the first identified object and the first unselectedobject.
 6. The system of claim 1, wherein the counter generates thecount based on a weighted sum of the scores.
 7. The system of claim 1,wherein the question is a natural language question.
 8. A methodcomprising: receiving, by a digital image processor, an image;identifying, by the digital image processor, objects in an image;embedding, by the digital image processor, the identified objects intoan embedding space; identifying, by the digital image processor, abounding box for each of the identified objects; receiving, by alanguage processor, a question; mapping, by the language processor, thequestion into the embedding space; determining, by a scorer, scores forthe identified objects, each respective score determining how well acorresponding one of the identified objects is responsive to thequestion; determining, by a counter, a count of the objects in the imagethat are responsive to the question based on the scores; and outputtingthe count and the bounding box for each object included in the count. 9.The method of claim 8, wherein determining the count comprises:determining whether to add a first identified object to the count;iteratively applying an interaction filter to remaining identifiedobjects to determine an uncounted object that is most likely responsiveto the question based on each previously counted object; with eachiteration, determining whether to add said uncounted object to thecount; and determining when to terminate counting objects in the image.10. The method of claim 9, wherein determining the first identifiedobject to count comprises identifying which of the identified objectshas a highest measure of match to the embedding of the question.
 11. Themethod of claim 10, wherein determining when to terminate processingobjects in the image comprises determining whether the highest measureof match is less than a termination value.
 12. The method of claim 9,further comprising determining the interaction filter for a firstunselected object of the identified objects based on the embedding ofthe question, the embedding corresponding to the first identifiedobject, the embedding corresponding to the first unselected object, andoverlaps between the bounding box of the first identified object and thebounding box of the first unselected object.
 13. The method of claim 8,wherein generating the count comprises determining a weighted sum of thescores.
 14. The method of claim 8, wherein the question is a naturallanguage question.
 15. A counter comprising: a logits module thatinitializes logit values for each of a plurality of candidate objectsbased on a scoring vector from a scorer, the scoring vector encoding howwell a plurality of candidate objects in an image match criteria in aquestion, each logit value indicating how likely a correspondingcandidate object is to be counted; an object selector that selects afirst object from the candidate objects that has a highest logit valueand counts the first object; and an interaction module that providesupdates to the logit values based on the selection of the first objectby the object selector.
 16. The counter of claim 15, wherein: the objectselector further iteratively selects one or more additional objects fromthe candidate objects based on their logit values; the interactionmodule further provides updates to the logit values with the selectionof each of the one or more additional objects; and the object selectorstops iteratively selecting the one or more additional objects whenlogit values of each of the candidate objects is less than a terminationvalue.
 17. The counter of claim 16, wherein the termination value istrainable.
 18. The counter of claim 15, further comprising: a weightingunit that receives the scoring vector from the scorer and appliestrainable weights to each score in the scoring vector; a summing unitthat adds a trainable bias to the weighted scoring vector; a transferfunction that is applied to an output of the summing unit and generatesinitial logits values for the logits module.
 19. The counter of claim15, wherein the interaction module determines an update to a first logitvalue of a first unselected object based on an embedding of thequestion, an object embedding corresponding to the first object, anobject embedding corresponding to the first unselected object, andoverlaps between bounding boxes of the first object and the firstunselected object.
 20. The counter of claim 15, wherein the objectselector further outputs a bounding box correspond to the first object.